1. Field
Embodiments of the invention relate generally to video compression, and more particularly, to a method and apparatus for frame rate up conversion for block-based low bit rate video.
2. Background
Low bit rate video compression is very important in many multimedia applications such as wireless video streaming and video telephony, due to the limited bandwidth resources and the variability of available bandwidth. Bandwidth adaptation video coding at low bit-rate can be accomplished by reducing the temporal resolution. In other words, instead of compressing and sending a 30 frames per second (fps) bit-stream, the temporal resolution can be halved to 15 fps to reduce the transmission bit-rate. However, the consequence of reducing temporal resolution is the introduction of temporal domain artifacts such as motion jerkiness that significantly degrades the visual quality of the decoded video.
To display the full frame rate at the receiver side, a recovery mechanism, called frame rate up conversion (FRUC), is needed to re-generate the skipped frames and to reduce temporal artifacts.
Many FRUC algorithms have been proposed, which can be classified into two categories. The first category interpolates the missing frame by using a combination of received video frames without taking the object motion into account. Frame repetition and frame averaging methods fit into this class. The drawbacks of these methods include the production of motion jerkiness, “ghost” images and blurring of moving objects when there is motion involved. The second category is more advanced, as compared to the first category, and utilizes the transmitted motion information, the so-called motion compensated (frame) interpolation (MCI).
As illustrated in prior art FIG. 1, in MCI, a missing frame 108 is interpolated based on a reconstructed current frame 102, a stored previous frame 104, and a set of transmitted motion vectors 106. Reconstructed current frame 102 is composed of a set of non-overlapped blocks 150, 152, 154 and 156 associated with set of transmitted motion vectors 106 pointing to corresponding blocks in stored previous frame 104. Interpolated frame 108 can be constructed in either a linear combination of corresponding pixels in current and previous frames; or nonlinear operation such as a median operation.
Block-based MCI introduces overlapped (multiple motion trajectories pass through this area) and hole (no motion trajectory passes through this area) regions in interpolated frames. As illustrated in FIG. 3, an interpolated frame 302 contains an overlapped area 306 and a hole area 304. The main reasons for the generation of these two types of unwanted areas are:
1. Moving objects are not under a rigid translational motion model.
2. The transmitted motion vectors used in the MCI may not point to the true motion trajectories due to the block-based fast motion search algorithms utilized in the encoder side.
3. The covered and uncovered background in the current frame and previous frames.
The interpolation of overlapped and hole regions is the major technical challenge in conventional block based motion compensated approaches. Median blurring and spatial interpolation techniques have been proposed to fill these overlapped and hole regions. However, the drawbacks of these methods are the introduction of the blurring and blocking artifacts, and also an increase in the complexity of interpolation operations.